Pointing device and cursor for use in intelligent computing environments

ABSTRACT

A system and process for directing a laser beam within a space is presented. The system includes a pointing device which periodically outputs orientation data indicative of the direction it is pointing and a cursor device which projects a laser beam. The orientation data is used to compute the direction the pointing device is pointing in terms of yaw and pitch angles. The laser beam is directed to locations in the space relative to the amount of movement of the pointing device. In an absolute mode, the pointed device and the laser beam of the cursor device are pointed at the same location in the space, whereas in a relative mode, the pointing device does not point at the same location as the laser beam. The position of the laser can be used to select a variety of hardware devices known to be in a room for future control actions.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of a prior application entitled “APOINTING DEVICE AND CURSOR FOR USE IN INTELLIGENT COMPUTINGENVIRONMENTS” which was assigned Ser. No. 10/461,646 and filed Jun. 13,2003.

BACKGROUND

1. Technical Field

The invention is related to cursor devices, and more particularly to asystem and process for directing a laser beam within a space to act as acursor in an intelligent computing environment.

2. Background Art

Ubiquitous (i.e., intelligent) computing promises to blur the boundariesbetween traditional desktop computing and the everyday physical world. Apopular vision of tomorrow's computing pushes computational abilitiesinto everyday objects, each participating in a complex and powerfulintegrated intelligent environment. Tomorrow's home and officeenvironments, for example, may include a variety of small and largenetworked displays and smart controllable devices. For instance, themodern living room typically features a television, amplifier, DVDplayer, lights, computers, and so on. In the near future, these deviceswill become more inter-connected, more numerous and more specialized aspart of an increasingly complex and powerful integrated intelligentenvironment.

This migration away from the desktop and “into the walls” presentsseveral challenges for user interface design. For example, how does theuser of tomorrow's intelligent environment select one of many devices?Today, this problem is most often addressed by maintaining a separateinterface, such as an IR remote control, for each device.

Tomorrow's intelligent environment presents the opportunity to present asingle intelligent user interface (UI) to control many such devices whenthey are networked. This UI device should provide the user a naturalinteraction with intelligent environments. For example, people havebecome quite accustomed to pointing at a piece of electronic equipmentthat they want to control, owing to the extensive use of IR remotecontrols. It has become almost second nature for a person in a modernenvironment to point at the object he or she wants to control, even whenit is not necessary. Take the small radio frequency (RF) key fobs thatare used to lock and unlock most automobiles in the past few years as anexample. Inevitably, a driver will point the free end of the key fobtoward the car while pressing the lock or unlock button. This is doneeven though the driver could just have well pointed the fob away fromthe car, or even pressed the button while still in his or her pocket,owing to the RF nature of the device. Thus, a single UI device, which ispointed at electronic components or some extension thereof (e.g., a wallswitch to control lighting in a room) to control these components, wouldrepresent an example of the aforementioned natural interaction that isdesirable for such a device.

There are some so-called “universal” remote controls on the market thatare preprogrammed with the known control protocols of a litany ofelectronic components, or which are designed to learn the commandprotocol of an electronic component. Typically, such devices are limitedto one transmission scheme, such as IR or RF, and so can control onlyelectronic components operating on that scheme. However, it would bedesirable if the electronic components themselves were passive in thatthey do not have to receive and process commands from the UI devicedirectly, but would instead rely solely on control inputs from theaforementioned network. In this way, the UI device does not have todifferentiate among various electronic components, say by recognizingthe component in some manner and transmitting commands using someencoding scheme applicable only to that component, as is the case withexisting universal remote controls.

Of course, a common control protocol could be implemented such that allthe controllable electronic components within an environment use thesame control protocol and transmission scheme. However, this wouldrequire all the electronic components to be customized to the protocoland transmission scheme, or to be modified to recognize the protocol andscheme. This could add considerably to the cost of a “singleUI-controlled” environment. It would be much more desirable if the UIdevice could be used to control any networked group of new or existingelectronic components regardless of remote control protocols ortransmission schemes the components were intended to operate under.

It is noted that in the remainder of this specification, the descriptionrefers to various individual publications identified by a numericdesignator contained within a pair of brackets. For example, such areference may be identified by reciting, “reference [1]” or simply“[1]”. A listing of references including the publications correspondingto each designator can be found at the end of the Detailed Descriptionsection.

SUMMARY

The present invention involves a system and process for directing alaser beam within a space to act as a cursor. This cursor device, whichwill be referred to as the WorldCursor, is analogous to the mouse andcursor used in traditional graphic user interfaces. Namely, a user mayselect and interact with a physical device by positioning the cursor onthe device and clicking. However, the WorldCursor goes much further. Itis a solution to providing a natural, expressive interface forinteraction in ubiquitous computing environments, where it is often arequirement to interact with devices beyond the desktop, and often inscenarios in which the traditional mouse and keyboard are inappropriateor unavailable. For example, the WorldCursor allows the user to point atand select items within a room, much as a mouse allows a user to pointat objects on a computer display.

The WorldCursor device itself includes a small platform that istypically installed on the ceiling. It has two small servo motors of thekind used in radio-controlled airplanes, and a laser, such as the redlasers used in laser pointing devices currently employed to givepresentations. The first of the servos is configured so as to move thelaser in a manner that controls the yaw direction of the laser beam andthe other servo is configured so as to move the laser in a manner thatcontrols the pitch direction of the laser beam. Thus, under computercontrol, the motors may be steered to point the laser almost anywhere inthe room.

One embodiment of the system employs control inputs from a conventionaldevice such as a computer mouse, trackball, gamepad, or the like todictate the movement of the WorldCursor laser. In this embodiment thecomputer controlling the movement of the laser receives movement controlcommands generated by one of the aforementioned movement control deviceswhich specifies the direction the laser beam is to be pointed. Thecomputer then provides commands to the WorldCursor device that directthe laser beam to move about the space as specified by the movementcontrol commands.

In another embodiment of the WorldCursor system, a pointing device isincluded that periodically outputs orientation data indicative of thedirection it is pointing. In addition, the WorldCursor and pointingdevices are both in communication with the aforementioned computer whichruns a program that receives the orientation data output by the pointingdevice and computes the direction the pointing device is pointing fromthe received orientation data in terms of yaw and pitch angles. Theprogram then directs the laser beam generated by the WorldCursor to aparticular location in the space as determined by the direction thepointing device is pointing.

The present system can further be employed to implement a process forselecting an object within a space. In general, this involves a usercausing the laser beam to shine on a selectable object by manipulatingthe pointing device and then using the device to select the object.

In tested versions of the WorldCursor system, the pointing device tookthe form of a hardware device referred to as the XWand, which is thesubject of a co-pending U.S. patent application entitled “A SYSTEM ANDPROCESS FOR SELECTING OBJECTS IN A UBIQUITOUS COMPUTING ENVIRONMENT”which was filed on May 31, 2002 and issued Ser. No. 10/160,692. TheXWand is a hardware device and software system which allows the user topoint at and operate various objects in the room. For example, the usermay point the XWand at a known light, push a button on the XWand, andthe light may turn on.

More particularly, the XWand device contains onboard sensors to supportthe computation of orientation information and gesture recognition.These sensors include a pair of accelerometers. When motionless, theseaccelerometers sense the acceleration due to gravity, and each can beused to sense either the pitch or roll angle of the device. Another ofthe sensors is a 3-axis magnetoresistive permalloy magnetometer. Thissenses the direction of the Earth's magnetic field in 3 dimensions, andcan be used to compute the yaw angle of the device. The values from theaccelerometer and magnetometer are relayed to a host computer by radiolink. These values are combined to find the absolute orientation of thedevice with respect to the room. This orientation is updated in realtime at a rate of about 50 Hz, and is accurate to a few degrees in eachof yaw, pitch and roll axes.

The XWand system determines which device the user is pointing at bycombining the orientation and 3-D position of the XWand with a 3-D modelof the room and the devices within it. Orientation of the XWand isdetermined as explained above from the onboard sensors, while XWandposition is determined with stereo computer vision. The 3-D model of theroom and devices is entered into the system by pointing with the XWanditself in a special training mode. With the orientation, position andmodel of the room, it is easy to determine which if any object in theworld model the XWand is pointing at. Audio feedback is provided toindicate to the user that the object is known to the system and can becontrolled by the XWand, but in general little feedback is necessarysince the pointing is absolute in nature.

The WorldCursor improves upon the XWand system by not requiring anexternal position sensing technology. There are a number of drawbacks tousing the external computer vision system. First, the system involvesinstalling multiple cameras in the room. Part of the installationrequires a rather precise calibration of the cameras against thegeometry of the room. Secondly, the acquisition of the geometric modelof the room and its devices requires a further calibration phase.Thirdly, computer vision techniques rely on having a clear line of sightto the device. Although this can be alleviated somewhat by installingmore cameras, this approach can be prohibitively expensive and complexto install. Finally, installation of cameras inevitably raises privacyobjections. The combination of the WorldCursor with the XWand eliminatesthe need for the external camera setup.

In a first mode of operation of the WorldCursor, the laser beam isdirected in an absolute pointing mode where the location that the laserbeam is pointed is substantially the same location that the pointingdevice (e.g., the XWand) is pointing. The process that accomplishes thisabsolute pointing mode involves first computing a set of offset anglesfor the laser made up of respective yaw and pitch angles that define theangular distance between the origin of a spherical coordinate systemassociated with the WorldCursor and a prescribed origin of the sphericalcoordinate system for the space. Likewise, a set of offset angles forthe pointing device are computed that represent the respective yaw andpitch angles defining the angular distance between the origin of aspherical coordinate system associated with the pointing device and theprescribed origin of the spherical coordinate system for the space.Next, aligning pitch and yaw angles are computed that define how far thelaser must be moved in order to point the laser beam at theapproximately the same point in the space that the pointing device ispointing. The aligning pitch angle is defined as the sum of the offsetpitch angle of the laser and the computed pitch angle of the pointingdevice less its offset pitch angle, and the aligning yaw angle isdefined as the sum of the offset yaw angle of the laser and the computedyaw angle of the pointing device less its offset yaw angle. Once thealigning angles are computed, the laser is moved to these angles so asto point the laser beam at approximately the same location in the spacethat the pointing device is pointing.

It is noted that the offset angles computed above are initial anglevalues which are typically only valid for a part of the space. If thepointing device and WorldCursor are very close to each other, the anglevalues would remain valid no matter where the pointing device isdirected. However, this will not be the usual case because theWorldCursor will typically be mounted on the ceiling of the space. Ifthe pointing device is pointed outside the part of the space where theinitial offset values are valid, the correspondence between the locationthe pointing device is pointing and the location the laser beam isshining is lost. Thus, the correspondence must be maintained in order tocontinue operating in the absolute pointing mode. If the user desiresthat the laser spot appear where the pointing device is pointed, or ifthe user would like to have the spot at least be in the field of theview of the user as they use the pointing device, a “clutching”procedure can be employed. To clutch the WorldCursor, the usermomentarily activates a switch on the pointing device (e.g., pushes itsbutton) whose state is included in each orientation data message sent bythe pointing device. When the switch is activated, the laser stopsmoving with the pointing device and the user then reorients the pointingdevice, lining it up so that it points directly at the laser spot. Whenready to resume WorldCursor control, the user reactivates the switch andthe laser spot beings moving again in correspondence with the pointingdevice.

It is noted that the user could also orient the pointing device during aclutching operation such that it does not point at the laser spot. Ifso, this creates a relative pointing condition where the laser mimicsthe movements of the pointing device (e.g., moving the XWand left orright produces the same direction of movement in the laser beam), butnot in correspondence with it. It is not clear if users require evenapproximate correspondence for successful use of the device. Experiencewith computer mice suggests that this is not necessary. Thus, thisrelative pointing mode may be of no consequence to the user, or evenpreferred.

One desirable feature of this system is that it does not require the 3Dposition of the pointing device to be known, nor a 3D model of the roomor of the devices within the room. Active devices such as lights,displays, etc., need only be known to the system by their spherical(yaw, pitch) coordinates which are used to set the position of the twomotors on the WorldCursor device. This can be accomplished using outsidemethods or by using the WorldCursor to generate a model of the space. Ingeneral, a model of the space can be accomplished by pointing the laserbeam at an object in the space and recording the pitch and yaw angles ofthe laser as its location. The user can then enter its extent in theform of a radius defining a circle that is used represent the object.Alternately, objects, such as walls, can be represented by polygons. Inthis case, the laser beam is made to shine on each vertex of the polygonrepresenting an object and the spherical coordinates of the laser arerecorded for each point. In either the circle or polygon cases, the usercan also enter information about the object, such as its identity thatcan be used to control the object once it is selected.

Once the space has been modeled as described above, it is possible todetermine whether the laser beam is shining on a modeled object so thatit can be selected for future control actions. In the case of a circlerepresenting an object in the space, the laser beam is considered to beon the object if the WorldCursor's laser pitch and yaw angles are withinthe specified radius of the circle. In the case of a polygonrepresenting an object, a conventional point-in-polygon technique can beuse to determine if the laser beam is on the object. Still further, itis possible to determine the point on a polygon representing an objectin the space that the laser beam is shining on using a transform thatmaps the pitch and yaw angle coordinates of the laser to orthogonalcoordinates of the polygon.

It is even possible to use the WorldCursor and the pointing device togenerate a 3D model of the space where objects are modeled as 3D pointswith extents or as polygons. More particularly, given the 3D location ofthe pointing device and its orientation in combination with the 3Dlocation of the WorldCursor and the orientation of its laser (which isassumed to be directed at the same point as the pointing device), then,it is possible to solve for unknown 3D point coordinates for anylocation the laser is shining on.

A procedure that can be employed to establish and maintain a reasonablecorrespondence between the pointing device and WorldCursor withoutclutching involves exploiting the geometry of the space in which theWorldCursor system is operating. If the geometry of the room is known interms of a 3D coordinate system, whether obtained from an outside sourceor generated as described above, as well as the 3D positions of theWorldCursor, and the pointing device, then the WorldCursor may becontrolled such that the pointing device always points directly at thelaser spot. It is noted that a reasonable correspondence between thelaser beam and the pointing device can be maintained even if the 3Dlocation of the pointing device is not known absolutely but is assumedto be at a location that it generally can be found.

However, if the approximate 3D geometry of the space is known orgenerated as described above, then there is also a technique to computethe precise 3D position of the pointing device (assuming it is unknownto the system) by exploiting the above mentioned clutching operation. Inthis way a more precise correspondence can be maintained as explainedabove. If the user is clutching so that the pointing device points atthe laser spot, each of these clutching operations provide informationthat can be related mathematically to the 3D position of the pointingdevice. After a few such clutching operations, it is possible to computethe 3D position of the device. The advantage in doing so is that no moreclutching operations are necessary to maintain pointingdevice-WorldCursor correspondence, so long as the actual position of thepointing device does not change dramatically. It is noted that thisonline calibration requires no more clutching operations than the systemwhich does not exploit the approximate room geometry. Further in thelong run it requires fewer clutching operations if the user does notmove about the room often.

In addition to the just described benefits, other advantages of thepresent invention will become apparent from the detailed descriptionwhich follows hereinafter when taken in conjunction with the drawingfigures which accompany it.

DESCRIPTION OF THE DRAWINGS

The specific features, aspects, and advantages of the present inventionwill become better understood with regard to the following description,appended claims, and accompanying drawings where:

FIG. 1 is a diagram depicting a general purpose computing deviceconstituting an exemplary system for implementing the present invention.

FIG. 2 is a diagram depicting a system for directing a laser beam withina space to act as a cursor according to the present invention.

FIG. 3 is an image depicting one prototype version of the WorldCursordevice employed in the system of FIG. 2.

FIGS. 4A and B are a flow chart diagramming a process for modelingobjects in a space using the WorldCursor system of FIG. 2, wherein theobjects are modeled as circles using spherical coordinates.

FIGS. 5A and B are a flow chart diagramming a process for modelingobjects in a space using the WorldCursor system of FIG. 2, wherein theobjects are modeled as polygons using spherical coordinates.

FIG. 6 is a flow chart diagramming a process for automatically switchingbetween slow and fast filters to adjust the relative cursormovement-to-pointing device movement speed.

FIGS. 7A and B are a flow chart diagramming a clutching process foraligning the WorldCursor laser beam and XWand by making the laser beamshine on approximately the same location in the space that the XWand ispointed.

FIG. 8 is a flow chart diagramming one version of a process forestablishing and maintain a reasonable alignment between the pointingdevice and WorldCursor involving computing the 3D location of thepointing device and using this along with a knowledge of the rest of thegeometry of the space to compute the pitch and yaw angles that whenapplied to the WorldCursor laser will make it point at approximately thesame location as the pointing device.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

In the following description of the preferred embodiments of the presentinvention, reference is made to the accompanying drawings which form apart hereof, and in which is shown by way of illustration specificembodiments in which the invention may be practiced. It is understoodthat other embodiments may be utilized and structural changes may bemade without departing from the scope of the present invention.

1.0 The Computing Environment

Before providing a description of the preferred embodiments of thepresent invention, a brief, general description of a suitable computingenvironment in which the invention may be implemented will be described.FIG. 1 illustrates an example of a suitable computing system environment100. The computing system environment 100 is only one example of asuitable computing environment and is not intended to suggest anylimitation as to the scope of use or functionality of the invention.Neither should the computing environment 100 be interpreted as havingany dependency or requirement relating to any one or combination ofcomponents illustrated in the exemplary operating environment 100.

The invention is operational with numerous other general purpose orspecial purpose computing system environments or configurations.Examples of well known computing systems, environments, and/orconfigurations that may be suitable for use with the invention include,but are not limited to, personal computers, server computers, hand-heldor laptop devices, multiprocessor systems, microprocessor-based systems,set top boxes, programmable consumer electronics, network PCs,minicomputers, mainframe computers, distributed computing environmentsthat include any of the above systems or devices, and the like.

The invention may be described in the general context ofcomputer-executable instructions, such as program modules, beingexecuted by a computer. Generally, program modules include routines,programs, objects, components, data structures, etc. that performparticular tasks or implement particular abstract data types. Theinvention may also be practiced in distributed computing environmentswhere tasks are performed by remote processing devices that are linkedthrough a communications network. In a distributed computingenvironment, program modules may be located in both local and remotecomputer storage media including memory storage devices.

With reference to FIG. 1, an exemplary system for implementing theinvention includes a general purpose computing device in the form of acomputer 110. Components of computer 110 may include, but are notlimited to, a processing unit 120, a system memory 130, and a system bus121 that couples various system components including the system memoryto the processing unit 120. The system bus 121 may be any of severaltypes of bus structures including a memory bus or memory controller, aperipheral bus, and a local bus using any of a variety of busarchitectures. By way of example, and not limitation, such architecturesinclude Industry Standard Architecture (ISA) bus, Micro ChannelArchitecture (MCA) bus, Enhanced ISA (EISA) bus, Video ElectronicsStandards Association (VESA) local bus, and Peripheral ComponentInterconnect (PCI) bus also known as Mezzanine bus.

Computer 110 typically includes a variety of computer readable media.Computer readable media can be any available media that can be accessedby computer 110 and includes both volatile and nonvolatile media,removable and non-removable media. By way of example, and notlimitation, computer readable media may comprise computer storage mediaand communication media. Computer storage media includes both volatileand nonvolatile, removable and non-removable media implemented in anymethod or technology for storage of information such as computerreadable instructions, data structures, program modules or other data.Computer storage media includes, but is not limited to, RAM, ROM,EEPROM, flash memory or other memory technology, CD-ROM, digitalversatile disks (DVD) or other optical disk storage, magnetic cassettes,magnetic tape, magnetic disk storage or other magnetic storage devices,or any other medium which can be used to store the desired informationand which can be accessed by computer 110. Communication media typicallyembodies computer readable instructions, data structures, programmodules or other data in a modulated data signal such as a carrier waveor other transport mechanism and includes any information deliverymedia. The term “modulated data signal” means a signal that has one ormore of its characteristics set or changed in such a manner as to encodeinformation in the signal. By way of example, and not limitation,communication media includes wired media such as a wired network ordirect-wired connection, and wireless media such as acoustic, RF,infrared and other wireless media. Combinations of the any of the aboveshould also be included within the scope of computer readable media.

The system memory 130 includes computer storage media in the form ofvolatile and/or nonvolatile memory such as read only memory (ROM) 131and random access memory (RAM) 132. A basic input/output system 133(BIOS), containing the basic routines that help to transfer informationbetween elements within computer 110, such as during start-up, istypically stored in ROM 131. RAM 132 typically contains data and/orprogram modules that are immediately accessible to and/or presentlybeing operated on by processing unit 120. By way of example, and notlimitation, FIG. 1 illustrates operating system 134, applicationprograms 135, other program modules 136, and program data 137.

The computer 110 may also include other removable/non-removable,volatile/nonvolatile computer storage media. By way of example only,FIG. 1 illustrates a hard disk drive 141 that reads from or writes tonon-removable, nonvolatile magnetic media, a magnetic disk drive 151that reads from or writes to a removable, nonvolatile magnetic disk 152,and an optical disk drive 155 that reads from or writes to a removable,nonvolatile optical disk 156 such as a CD ROM or other optical media.Other removable/non-removable, volatile/nonvolatile computer storagemedia that can be used in the exemplary operating environment include,but are not limited to, magnetic tape cassettes, flash memory cards,digital versatile disks, digital video tape, solid state RAM, solidstate ROM, and the like. The hard disk drive 141 is typically connectedto the system bus 121 through an non-removable memory interface such asinterface 140, and magnetic disk drive 151 and optical disk drive 155are typically connected to the system bus 121 by a removable memoryinterface, such as interface 150.

The drives and their associated computer storage media discussed aboveand illustrated in FIG. 1, provide storage of computer readableinstructions, data structures, program modules and other data for thecomputer 110. In FIG. 1, for example, hard disk drive 141 is illustratedas storing operating system 144, application programs 145, other programmodules 146, and program data 147. Note that these components can eitherbe the same as or different from operating system 134, applicationprograms 135, other program modules 136, and program data 137. Operatingsystem 144, application programs 145, other program modules 146, andprogram data 147 are given different numbers here to illustrate that, ata minimum, they are different copies. A user may enter commands andinformation into the computer 110 through input devices such as akeyboard 162 and pointing device 161, commonly referred to as a mouse,trackball or touch pad. Other input devices (not shown) may include amicrophone, joystick, game pad, satellite dish, scanner, or the like.These and other input devices are often connected to the processing unit120 through a user input interface 160 that is coupled to the system bus121, but may be connected by other interface and bus structures, such asa parallel port, game port or a universal serial bus (USB). A monitor191 or other type of display device is also connected to the system bus121 via an interface, such as a video interface 190. In addition to themonitor, computers may also include other peripheral output devices suchas speakers 197 and printer 196, which may be connected through anoutput peripheral interface 195. Of particular significance to thepresent invention, a camera 163 (such as a digital/electronic still orvideo camera, or film/photographic scanner) capable of capturing asequence of images 164 can also be included as an input device to thepersonal computer 110. Further, while just one camera is depicted,multiple cameras could be included as input devices to the personalcomputer 110. The images 164 from the one or more cameras are input intothe computer 110 via an appropriate camera interface 165. This interface165 is connected to the system bus 121, thereby allowing the images tobe routed to and stored in the RAM 132, or one of the other data storagedevices associated with the computer 110. However, it is noted thatimage data can be input into the computer 110 from any of theaforementioned computer-readable media as well, without requiring theuse of the camera 163.

The computer 110 may operate in a networked environment using logicalconnections to one or more remote computers, such as a remote computer180. The remote computer 180 may be a personal computer, a server, arouter, a network PC, a peer device or other common network node, andtypically includes many or all of the elements described above relativeto the computer 110, although only a memory storage device 181 has beenillustrated in FIG. 1. The logical connections depicted in FIG. 1include a local area network (LAN) 171 and a wide area network (WAN)173, but may also include other networks. Such networking environmentsare commonplace in offices, enterprise-wide computer networks, intranetsand the Internet.

When used in a LAN networking environment, the computer 110 is connectedto the LAN 171 through a network interface or adapter 170. When used ina WAN networking environment, the computer 110 typically includes amodem 172 or other means for establishing communications over the WAN173, such as the Internet. The modem 172, which may be internal orexternal, may be connected to the system bus 121 via the user inputinterface 160, or other appropriate mechanism. In a networkedenvironment, program modules depicted relative to the computer 110, orportions thereof, may be stored in the remote memory storage device. Byway of example, and not limitation, FIG. 1 illustrates remoteapplication programs 185 as residing on memory device 181. It will beappreciated that the network connections shown are exemplary and othermeans of establishing a communications link between the computers may beused.

The exemplary operating environment having now been discussed, theremaining part of this description section will be devoted to adescription of the program modules embodying the invention.

2.0 The XWand System

In a co-pending U.S. patent application entitled “A SYSTEM AND PROCESSFOR SELECTING OBJECTS IN A UBIQUITOUS COMPUTING ENVIRONMENT” which wasfiled on May 31, 2002 and issued Ser. No. 10/160,692, a system andprocess was introduced that provides a remote control UI device that iscapable of controlling a group of networked electronic components. Moreparticularly, the UI device, which will herein be referred to as theXWand, is able to control the electronic components without having todirectly differentiate among the components or employ a myriad ofdifferent control protocols and transmission schemes. And in order toprovide a natural interaction experience, the present system is operatedby having the user point at the electronic component (or an extensionthereof) that he or she wishes to control. In particular, the XWandprovides a remote control UI device that can be simply pointed atobjects in an ubiquitous computing environment that are associated insome way with controllable, networked electronic components, so as toselect that object for controlling via the network. This can for exampleinvolve pointing the UI device at a wall switch and pressing a button onthe device to turn a light operated by the switch on or off. The idea isto have a UI device so simple that it requires no particular instructionor special knowledge on the part of the user.

The XWand system includes the aforementioned remote control UI device inthe form of a wireless RF pointer, which includes a radio frequency (RF)transceiver and various orientation sensors. The outputs of the sensorsare periodically packaged as orientation messages and transmitted usingthe RF transceiver to a base station, which also has a RF transceiver toreceive the orientation messages transmitted by the pointer. There isalso a pair of digital video cameras each of which is located so as tocapture images of the environment in which the pointer is operating fromdifferent viewpoints. A computer, such as a PC, is connected to the basestation and the video cameras. Orientation messages received by the basestation from the pointer are forwarded to the computer, as are imagescaptured by the video cameras. The computer is employed to compute theorientation and location of the pointer using the orientation messagesand captured images. The orientation and location of the pointer is inturn used to determine if the pointer is being pointed at an object inthe environment that is controllable by the computer via a networkconnection. If it is, the object is selected.

The pointer specifically includes a case having a shape with a definedpointing end, a microcontroller, the aforementioned RF transceiver andorientation sensors which are connected to the microcontroller, and apower supply (e.g., batteries) for powering these electronic components.In the tested versions of the pointer, the orientation sensors includedat least, an accelerometer that provides separate x-axis and y-axisorientation signals, and a magnetometer that provides separate x-axis,y-axis and z-axis orientation signals. These electronics were housed ina case that resembled a wand—hence the XWand name.

The pointer's microcontroller packages and transmits orientationmessages at a prescribed rate. While the microcontroller could beprogrammed to accomplish this task by itself, a command-responseprotocol was employed in tested versions of the system. This entailedthe computer periodically instructing the pointer's microcontroller topackage and transmit an orientation message by causing the base stationto transmit a request for the message to the pointer at the prescribedrate. This prescribed rate could for example be approximately 50 timesper second as it was in tested versions of the system.

As indicated previously, the orientation messages generated by thepointer include the outputs of the sensors. To this end, the pointer'smicrocontroller periodically reads and stores the outputs of theorientation sensors. Whenever a request for an orientation message isreceived (or it is time to generate such a message if the pointer isprogrammed to do so without a request), the microcontroller includes thelast-read outputs from the accelerometer and magnetometer in theorientation message.

The pointer also includes other electronic components such as a useractivated switch or button, and a series of light emitting diodes(LEDs). The user-activated switch, which is also connected to themicrocontroller, is employed for the purpose of instructing the computerto implement a particular function. To this end, the state of the switchin regard to whether it is activated or deactivated at the time anorientation message is packaged is included in that message fortransmission to the computer. The series of LEDs includes a pair ofdifferently-colored, visible spectrum LEDs, which are connected to themicrocontroller, and which are visible from the outside of the pointer'scase when lit. These LEDs are used to provide status or feedbackinformation to the user, and are controlled via instructions transmittedto the pointer by the computer.

The foregoing system is used to select an object by having the usersimply point to the object with the pointer. This entails the computerfirst inputting the orientation messages transmitted by the pointer. Foreach message received, the computer derives the orientation of thepointer in relation to a predefined coordinate system of the environmentin which the pointer is operating using the orientation sensor readingscontained in the message. In addition, the video output from the videocameras is used to ascertain the location of the pointer at a timesubstantially contemporaneous with the generation of the orientationmessage and in terms of the predefined coordinate system. Once theorientation and location of the pointer are computed, they are used todetermine whether the pointer is being pointed at an object in theenvironment that is controllable by the computer. If so, then thatobject is selected for future control actions.

The computer derives the orientation of the pointer from the orientationsensor readings contained in the orientation message as follows. First,the accelerometer and magnetometer output values contained in theorientation message are normalized. Angles defining the pitch of thepointer about the x-axis and the roll of the device about the y-axis arecomputed from the normalized outputs of the accelerometer. Thenormalized magnetometer output values are then refined using these pitchand roll angles. Next, previously established correction factors foreach axis of the magnetometer, which relate the magnetometer outputs tothe predefined coordinate system of the environment, are applied to theassociated refined and normalized outputs of the magnetometer. The yawangle of the pointer about the z axis is computed using the refinedmagnetometer output values. The computed pitch, roll and yaw angles arethen tentatively designated as defining the orientation of the pointerat the time the orientation message was generated. It is next determinedwhether the pointer was in a right-side up or up-side down position atthe time the orientation message was generated. If the pointer was inthe right-side up position, the previously computed pitch, roll and yawangles are designated as the defining the finalized orientation of thepointer. However, if it is determined that the pointer was in theup-side down position at the time the orientation message was generated,the tentatively designated roll angle is corrected accordingly, and thenthe pitch, yaw and modified roll angle are designated as defining thefinalized orientation of the pointer. In the foregoing description, itis assumed that the accelerometer and magnetometer of the pointer areoriented such that their respective first axis corresponds to the x-axiswhich is directed laterally to a pointing axis of the pointer and theirrespective second axis corresponds to the y-axis which is directed alongthe pointing axis of the pointer, and the third axis of the magnetometercorrespond to the z-axis which is directed vertically upward when thepointer is positioned right-side up with the x and y axes lying in ahorizontal plane.

The computer derives the location of the pointer from the video outputof the video cameras as follows. There is an infrared (IR) LED connectedto the microcontroller that is able to emit IR light outside thepointer's case when lit. The microcontroller causes the IR LEDs toflash. In addition, the aforementioned pair of digital video cameraseach have an IR pass filter that results in the video image framescapturing only IR light emitted or reflected in the environment towardthe camera, including the flashing from the pointer's IR LED whichappears as a bright spot in the video image frames. The microcontrollercauses the IR LED to flash at a prescribed rate that is approximatelyone-half the frame rate of the video cameras. This results in only oneof each pair of image frames produced by a camera having the IR LEDflashes depicted in it. This allows each pair of frames produced by acamera to be subtracted to produce a difference image, which depicts forthe most part only the IR emissions and reflections directed toward thecamera which appear in one or the other of the pair of frames but notboth (such as the flash from the IR LED of the pointing device). In thisway, the background IR in the environment is attenuated and the IR flashbecomes the predominant feature in the difference image. The imagecoordinates of the pixel in the difference image that exhibits thehighest intensity is then identified using a standard peak detectionprocedure. A conventional stereo image technique is then employed tocompute the 3D coordinates of the flash for each set of approximatelycontemporaneous pairs of image frames generated by the pair of camerasusing the image coordinates of the flash from the associated differenceimages and predetermined intrinsic and extrinsic camera parameters.These coordinates represent the location of the pointer (as representedby the location of the IR LED) at the time the video image frames usedto compute them were generated by the cameras.

The orientation and location of the pointing device at any given time isused to determine whether the pointing device is being pointed at anobject in the environment that is controllable by the computer. In orderto do so the computer must know what objects are controllable and wherethey exist in the environment. This requires a model of the environment.In the XWand system, the location and extent of objects within theenvironment that are controllable by the computer are modeled using 3DGaussian blobs defined by a location of the mean of the blob in terms ofits environmental coordinates and a covariance. Two different methodshave been developed to model objects in the environment.

The first involves the user inputting information identifying the objectthat is to be modeled. The user then activates the switch on thepointing device and traces the outline of the object. Meanwhile, thecomputer is running a target training procedure that causes requests fororientation messages to be sent to the pointing device a prescribedrequest rate. The orientation messages are input as they are received,and for each orientation message, it is determined whether the switchstate indicator included in the orientation message indicates that theswitch is activated. Whenever it is initially determined that the switchis not activated, the switch state determination action is repeated foreach subsequent orientation message received until an orientationmessage is received which indicates that the switch is activated. Atthat point, each time it is determined that the switch is activated, thelocation of the pointing device is ascertained as described previouslyusing the digital video input from the pair of video cameras. When theuser is done tracing the outline of the object being modeled, he or shedeactivates the switch. The target training process sees this as theswitch has been deactivated after having been activated in theimmediately preceding orientation message. Whenever such a conditionoccurs, the tracing procedure is deemed to be complete and a 3D Gaussianblob representing the object is established using the previouslyascertained pointing device locations stored during the tracingprocedure.

The second method of modeling objects once again begins by the userinputting information identifying the object that is to be modeled.However, in this case the user repeatedly points the pointer at theobject and momentarily activates the switch on the device, each timepointing the device from a different location within the environment.Meanwhile, the computer is running a target training procedure thatcauses requests for orientation messages to be sent to the pointingdevice at a prescribed request rate. Each orientation message receivedfrom the pointing device is input until the user indicates the targettraining inputs are complete. For each orientation message input, it isdetermined whether the switch state indicator contained thereinindicates that the switch is activated. Whenever it is determined thatthe switch is activated, the orientation of the pointing device iscomputed as described previously using orientation sensor readings alsoincluded in the orientation message. In addition, the location of thepointing device is ascertained using the inputted digital video from thepair of video cameras. The computed orientation and location values arestored. Once the user indicates the target training inputs are complete,the location of the mean of a 3D Gaussian blob that will be used torepresent the object being modeled is computed from the pointingdevice's stored orientation and location values. The covariance of theGaussian blob is then obtained in one of various ways. For example, itcan be a prescribed covariance, a user input covariance, or thecovariance can be computed by adding a minimum covariance to the spreadof the intersection points of rays defined by the pointing device'sstored orientation and location values.

With a Gaussian blob model of the environment in place, the orientationand location of the pointing device can be is used to determine whetherthe pointing device is being pointed at an object in the environmentthat is controllable by the computer. In one version of this procedure,for each Gaussian blob in the model, the blob is projected onto a planewhich is normal to either a line extending from the location of thepointing device to the mean of the blob or a ray originating at thelocation of the pointing device and extending in a direction defined bythe orientation of the device. The value of the resulting projectedGaussian blob at a point where the ray intersects the plane is computed.This value represents the probability that the pointing device ispointing at the object associated with the blob under consideration.Next, the probability representing the largest value computed for theGaussian blobs, if any, is identified. At this point, the objectassociated with the Gaussian blob from which the largest probabilityvalue was derived could be designated as being the object that thepointing device is pointing at. However, an alternate thresholdingprocedure could be employed instead. In this alternate version, it isfirst determined whether the probability value identified as the largestexceeds a prescribed minimum probability threshold. Only if thethreshold is exceeded is the object associated with the projectedGaussian blob from which the largest probability value was deriveddesignated as being the object that the pointer is pointing at. Theminimum probability threshold is chosen to ensure the user is actuallypointing at the object and not just near the object without an intent toselect it.

In an alternate procedure for determining whether the pointing device isbeing pointed at an object in the environment that is controllable bythe computer, for each Gaussian blob, it is determined whether a rayoriginating at the location of the pointing device and extending in adirection defined by the orientation of the device intersects the blob.Next, for each Gaussian blob intersected by the ray, it is determinedwhat the value of the Gaussian blob is at a point along the ray nearestthe location of the mean of the blob. This value represents theprobability that the pointing device is pointing at the objectassociated with the Gaussian blob. The rest of the procedure is similarto the first method in that the object associated with the Gaussian blobfrom which the largest probability value was derived could be designatedas being the object that the pointing device is pointing at. Oralternately, it is first determined whether the probability valueidentified as the largest exceeds the prescribed minimum probabilitythreshold. If the threshold is exceeded, only then is the objectassociated with the projected Gaussian blob from which the largestprobability value was derived designated as being the object that thepointing device is pointing at.

Users of the XWand are often impressed with the immediate and naturalfeel of absolute pointing. However, the pure geometry-based approachwhich enables absolute pointing also has a number of importantdrawbacks. First, two or more cameras must be permanently mounted in theroom. Besides the difficulty of installation, such cameras inevitablydraw objections related to privacy. In addition, the cameras must becarefully calibrated to the room geometry upon installation, andrecalibrated if they are moved. Further, at least two cameras must haveclear sight-lines to the wand at all times. Finally, the threedimensional position of each active device in the room must be known,and small errors in the orientation and position information translateto inaccuracy in pointing, possibly disrupting the interaction.

Given these objections, alternatives to absolute pointing would beadvantageous with the goal of eliminating the three dimensionalpositioning system. One general approach is to place tags in theenvironment, but they have drawbacks as well. By design tags requireinstallation on every active device. Active tags such as IR beacons, forexample, require their own power, while passive tags such as RF ID tagstend to have limited range, and tags based on visual features rely onrather sophisticated onboard processing.

3.0 WorldCursor System

The foregoing XWand system issues are resolved by the present system,which will be referred to herein as the WorldCursor system. TheWorldCursor system uses the XWand device (or similar pointing device)but does not rely on a geometric model of pointing that requires thethree dimensional position of the XWand, nor on tags placed in theenvironment, nor on any external sensing in general. Instead, a laserbeam projected in the space gives the user feedback as to where thesystem believes the user is pointing, much in the same way that thecursor icon in “windows, icons, menus and pointing” (WIMP) interfacesprovides feedback to indicate where the user is pointing with the mouse.In fact, the WorldCursor is analogous to the mouse and cursor used intraditional GUIs in that the user may select and interact with aphysical device by positioning the cursor on the device and clicking.

In the foregoing context, the XWand is employed as a physical pointingmechanism, and it is coupled with the WorldCursor which projects acursor on the physical environment. The WorldCursor improves upon theXWand by removing the need for external positioning technology such asvideo cameras or any other external position sensing technology, and byenabling the user to point with a high degree of precision.

Referring to FIG. 2, the WorldCursor system includes a smalltele-operated motion platform 200 upon which is mounted a laser pointer.This device is controlled via a wired connection 202 to a host computer204, which is also connected to the XWand RF base station 206. TheWorldCursor platform 200 can be programmed to follow the motion of theXWand 208, such that when the user points the XWand to the left, forexample, the WorldCursor moves a corresponding amount to the left inreal time. The user attends to the projected laser spot (the cursor) inthe environment. By moving the XWand the user is then able to place thecursor on any object in the room, as they would place the cursor on anonscreen object with the mouse. Because only the orientation informationfrom the XWand is used, and not the XWand's 3-D position, the originalXWand system's requirement of the external computer vision system iseliminated.

Interacting with active devices in the intelligent environment proceedsmuch as in the original XWand system. For example, to turn a householdlamp on or off, instead of pointing directly at the lamp, the user movesthe laser spot onto the lamp and clicks the XWand button. The systemdetermines that the cursor is on the lamp by comparing the currentcursor position with the recorded cursor position associated with thelamp, collected beforehand.

3.1 The WorldCursor Device

In general, the WorldCursor device simply needs to take yaw and pitchcommands in some form and in response move the laser spot to any desiredplace (within line of sight of the laser) in the space in which it isoperating. Any device having this capability will suffice for use in theoverall WorldCursor system. In tested embodiments of the WorldCursorsystem the aforementioned device took the form of a motion platform thatis mounted on the ceiling, typically near the center of the room. Aprototype of this device is shown in FIG. 3. It consisted of two highspeed miniature servos 300, 302, such as the type used onradio-controlled model airplanes. Specifically, in tested embodiments ofthe WorldCursor, Expert Electronic's SL451 High Speed Mini Servos (ModelEXRSL451) were used. One of the servos 300 was mounted for yaw and asecond of the servos 302 for pitch control. In one embodiment of thedevice, both servos 300, 302 were controlled by a PIC microcontroller,which takes yaw and pitch commands from the host computer via a RS-232connection and converted them to standard servo motor commands. In analternate embodiment, an API on the aforementioned host computer takesyaw and pitch values from the XWand, and converts then to standard servomotor commands. These commands are then sent to the servos 300, 302. Itis noted that this latter scenario has the advantage of sending motorcommands that are typically in integer form, rather than floating pointdata such as would be the case with pitch and yaw values.

Mounted on the servo assembly is a red laser 304 similar to those usedin conventional laser pointers. By controlling the servos 300, 302, theplatform is able to steer the laser spot to most points in the roombelow the ceiling, provided there exists a sight line to that point. Intested versions of the present system and process, effective resolutionin steering the laser using the aforementioned servos is about 0.25degrees or about one half inch at a distance of 9 feet. The servos wereeach capable of moving over nearly a 170 degree range at a speed of 333degrees per second. Generally, this configuration resulted in the motionof the laser being smooth and responsive. However, in the case when thelaser must move from pointing to a location in front of the platform toa location behind the platform, the pitch motor must move to the backand the yaw motor must reflect about the vertical plane separating thefront and rear hemispheres. Because the servos employed in the testedembodiments had a 170 degree range limitation, there was a discontinuityin this movement of the laser spot from front to back (i.e., a 20 degreegap at the sides). While the aforementioned pointing inaccuracy anddiscontinuity were not found to be a problem in the tested embodiments,ideally, servos with a full 180 degree range and higher accuracy couldbe employed to resolve these minor deficiencies.

It is noted that the connection between the WorldCursor base unit andthe host computer could also be of a wireless type. However, if this isthe case care must be taken to ensure there is no interference with theXWand system.

3.2 World Model for the WorldCursor System

The WorldCursor points at a given object in the room by changing thepitch and yaw of the laser with its motors. It is therefore possible touniquely associate a given object in the room with the yaw and pitchvalue used to point the WorldCursor at the object. The yaw and pitchvalues of each object of interest in the space are the basis for aconvenient world model for the WorldCursor system based on sphericalcoordinates.

The spherical coordinate world model is easier to construct than thefull three dimensional model of the original XWand system as describedpreviously. For example, whereas in the three dimensional model the userhad to either hold the XWand over the object, or provide severalpointing examples used to triangulate the position of the object, theWorldCursor system need only record the current yaw and pitch values ofthe device once the user has put the cursor on the object. Onelimitation of this approach is that the spherical coordinate world modelmust be re-learned if the WorldCursor device is moved to a new location.

Given this, a model of the space that the WorldCursor system is tooperate in can be established as follows. Referring to FIGS. 4A and B,the user initiates a training mode that is part of a WorldCursor controlprocess running on the host computer (process action 400). The trainingmode has the same purpose as a similar process used in the XWandsystem—namely to learn where objects of interest are in the space. Moreparticularly, the user directs the laser at an object of interest bypointing the XWand so that the laser spot appears on the approximatecenter of the object being modeled and presses the button on the XWand(process action 402). The control process causes periodic requests to besent to the XWand directing it to provide an orientation message in themanner described previously (process action 404). Any incomingorientation message transmitted by the pointer is input (process action406), and it is determined whether the button state indicator includedin the message indicates that the pointer's button is activated (processaction 408). If not, process actions 406 and 408 are repeated. When itis discovered that the button state indicator indicates the button isactivated, then in process action 412, the control process accepts inputfrom the user who enters information into the host computer thatidentifies the object being modeled, including its approximate size(process action 410). Since it was the control process running on thehost computer that received the pitch and yaw data from the XWand andmoved the laser of the WorldCursor to the target location as describedpreviously, the pitch and yaw associated with the target spot are known.The process associates the spherical coordinates of the target locationto the information entered by the user about the corresponding object(process action 414). In addition, the user-provided size data is usedto establish a circle in spherical coordinates about the direction thelaser is pointed that models the extent of the object (process action416). Any incoming orientation message transmitted by the pointercontinues to be input (process action 418) and a determination is madeas to whether the button state indicator included in the messages firstindicates that the pointer's button becomes deactivated and thenactivated again, thereby indicating that the user has pushed the XWandbutton again (process action 420). If it is, process actions 418 and 420are repeated. If it is not, in process action 424, it is determined ifthe user has deactivated the training mode (process action 422), thusindicating all the objects it is desired to model in the space have beenmodeled. If the user has not deactivated the training mode, then processactions 402 through 424 are repeated to “learn” the next object the userchooses to identify to the system. Otherwise the process ends.

Once the objects in the space have been modeled as described above, usercan direct the laser of the WorldCursor to the modeled objects and actupon them as was done in the XWand system. More particularly, the usershines the WorldCursor's laser beam on the object he or she wants toselect. It is then determined whether the laser beam is on an object inthe space that is known. Whenever the laser beam is on a known object,that object is selected for future control actions.

To determine if the laser beam is on an object, the user is required toactivate the XWand switch when the laser beam is shining on the objecthe or she wants to select. The distance is computed in sphericalcoordinates between the current WorldCursor position and a positionstored for each of the modeled objects. For coordinates (θ_(c), φ_(c))of the WorldCursor and coordinates (θ_(i), φ_(i)) of the center of agiven object, the user is deemed to be pointing at a modeled object if:√{square root over ((θ_(i)−θ_(c))²+(φ_(i)−φ_(c))²)}<r _(i)   (1)where radius r_(i) indicates the size of the object modeled as a circlein spherical coordinates. It is noted that this method of determining ifthe user is pointing at a modeled object in the environment is clearlymuch easier than the previously-described Gaussian blob technique usedin connection with the standalone XWand system.

In some cases the object being modeled in the environment will be betterrepresented as a polygon rather than a circle. In addition, in somecases the WorldCursor may be needed to indicate one or more points on anobject with a high degree of precision. Both of these issues areresolved by modeling the object in question as a polygon. This isaccomplished by inputting a set of vertices that form a polygonal modelof an object. To input the vertices, the user places the laser spot ofthe WorldCursor on each vertex of the polygon representing the object,in turn, while the WorldCursor system is in a training mode. Once theobject is “trained”, the system can then determine if the cursor is onthe polygon by using standard point-in-polygon algorithms used in twodimensional graphics [1].

More particularly, a model of a polygonal object in the space that theWorldCursor system is operating in can be established as follows.Referring to FIGS. 5A and B, the user initiates the aforementionedtraining mode that is part of an control process running on the hostcomputer, except also indicating a polygonal object is being modeled(process action 500). The user directs the laser by pointing the XWandso that the laser spot appears one of the vertices of the object beingmodeled and presses the button on the XWand (process action 502). Theprocedure then proceeds as before with the control process causingperiodic requests to be sent to the XWand directing it to provide anorientation message in the manner described previously (process action504). Any incoming orientation message transmitted by the pointer isinput (process action 506), and it is determined whether the buttonstate indicator included in the message indicates that the pointer'sbutton is activated (process action 508). If not, process actions 506and 508 are repeated. When it is discovered that the button stateindicator indicates the button is activated, the process associates thespherical coordinates of the target location to the object vertex beingmodeled (process action 510). Any incoming orientation messagetransmitted by the pointer continues to be input (process action 512)and a determination is made as to whether the button state indicatorincluded in the messages first indicates that the pointer's buttonbecomes deactivated and then activated again, thereby indicating thatthe user has pushed the XWand button again (process action 514). If itis, process actions 512 and 514 are repeated. If it is not, in processaction 518, it is determined if the last vertex has been identified bythe user (process action 516). If so the process ends. If not, processactions 502 through 518 are repeated to “learn” the next vertex of thepolygon. Optionally, either before or after the user points out all thevertices of the polygon being established to the model the object ofinterest, he or she can enter information into the host computer thatidentifies the object. This information would be associated with theobject as well.

Once an object has been modeled as a polygon, it might also be desirableto know precisely where the cursor (i.e., laser spot) is on the object'ssurface in a local coordinate system, as mentioned above. For example,this polygon technique can be used to determine if the WorldCursor spotis on the active surface of a computer display, and if so where on thatsurface. In this way the WorldCursor can act as a display cursor aswell.

For a four vertex polygonal surface such as a computer display, thedesired location would be the cursor's screen coordinates. A transformfrom 25 WorldCursor to screen coordinates allows the display to beseamlessly incorporated into the rest of the world model, as describedlater. In this case, a projective transform [3] can be used to transformWorldCursor coordinates to screen coordinates (x, y) as follows:$\begin{matrix}{\begin{bmatrix}{wx} \\{wy} \\w\end{bmatrix} = {\begin{bmatrix}p_{11} & p_{12} & p_{13} \\p_{21} & p_{22} & p_{23} \\p_{31} & p_{32} & 1\end{bmatrix}\begin{bmatrix}\theta_{c} \\\phi_{c} \\1\end{bmatrix}}} & (2)\end{matrix}$The parameters p_(ij) are determined by solving a linear system ofequations given the four corners of the display in both WorldCursor andscreen coordinates. Here the assumption is made that the WorldCursorcoordinate system is linear in the region of the display, even thoughmodeled in spherical coordinates—a valid assumption for typical sizeddisplays.

More particularly, the projective transform of Eq. (2) takes coordinates(x,y) into coordinates (x′,y′) by way of a 3×3 matrix with 8 freeparameters. Points (x_(i),y_(i)) are the coordinates of the 4 corners ofthe polygon in WorldCursor device coordinates, and the points(x′_(i),y′_(i)) are the coordinates of the same 4 corners in localcoordinates (e.g., screen coordinates of a computer display). Thesevalues are collected in an offline training procedure in which thecursor is placed on each of the 4 corners of the polygon in turn, asdescribed previously.The matrix may be expressed as $\begin{matrix}{{x^{\prime} = \frac{{p_{11}x} + {p_{12}y} + p_{13}}{{p_{31}x} + {p_{32}y} + 1}},{y^{\prime} = {\frac{{p_{21}x} + {p_{22}y} + p_{23}}{{p_{31}x} + {p_{32}y} + 1}.}}} & (3)\end{matrix}$Rearranging terms gives:x′=p ₁₁ x+p ₁₂ y+p ₁₃ −p ₃₁ xx′−p ₃₂ yx′y′=p ₂₁ x+p ₂₂ y+p ₂₃ −p ₃₁ xy′−p ₃₂ yy′  (4)From this linear system it is possible to solve for parameters p_(ij)given 4 points (x_(i),y_(i)) which map to corresponding points(x′_(i),y′_(i)): $\begin{matrix}{\begin{bmatrix}x_{1}^{\prime} \\y_{1}^{\prime} \\x_{2}^{\prime} \\y_{2}^{\prime} \\x_{3}^{\prime} \\y_{3}^{\prime} \\x_{4}^{\prime} \\y_{4}^{\prime}\end{bmatrix} = {\begin{bmatrix}x_{1} & y_{1} & 1 & 0 & 0 & 0 & {{- x_{1}^{\prime}}x_{1}} & {{- x_{1}^{\prime}}y_{1}} \\0 & 0 & 0 & x_{1} & y_{1} & 1 & {{- y_{1}^{\prime}}x_{1}} & {{- y_{1}^{\prime}}y_{1}} \\x_{2} & y_{2} & 1 & 0 & 0 & 0 & {{- x_{2}^{\prime}}x_{2}} & {{- x_{2}^{\prime}}y_{2}} \\0 & 0 & 0 & x_{2} & y_{2} & 1 & {{- y_{2}^{\prime}}x_{2}} & {{- y_{2}^{\prime}}y_{2}} \\x_{3} & y_{3} & 1 & 0 & 0 & 0 & {{- x_{3}^{\prime}}x_{3}} & {{- x_{3}^{\prime}}y_{3}} \\0 & 0 & 0 & x_{3} & y_{3} & 1 & {{- y_{3}^{\prime}}x_{3}} & {{- y_{3}^{\prime}}y_{3}} \\x_{4} & y_{4} & 1 & 0 & 0 & 0 & {{- x_{4}^{\prime}}x_{4}} & {{- x_{4}^{\prime}}y_{4}} \\0 & 0 & 0 & x_{4} & y_{4} & 1 & {{- y_{4}^{\prime}}x_{4}} & {{- y_{4}^{\prime}}y_{4}}\end{bmatrix}\begin{bmatrix}p_{11} \\p_{12} \\p_{13} \\p_{21} \\p_{22} \\p_{23} \\p_{31} \\p_{32}\end{bmatrix}}} & (5)\end{matrix}$During runtime, screen coordinates are computed from WorldCursor devicecoordinates using equation (2) above. Note that this computation is onlyperformed after it is determined that the cursor is contained within thepolygon described by the points (x_(i),y_(i)). Alternatively, one mayalways perform this mapping, and then check if the resulting screencoordinate values are contained within the polygon described by thepoints (x′_(i),y′_(i)) (a trivial calculation).3.3 Controlling the WorldCursor System

In this section, various techniques for controlling the WorldCursor withthe XWand are presented. The general scheme of the control mapping isthat the WorldCursor should mimic the exact motion of the XWand, much inthe same way that the cursor on a WIMP interface mimics the motion ofthe mouse. If (θ_(c), φ_(c)) is the yaw and pitch of the WorldCursor and(θ_(w), φ_(w)) is the yaw and pitch of the XWand, then (θ_(c),φ_(c))≈(θ_(w), φ_(w)), at least for an absolute pointing mode as will bedescribed shortly.

3.3.1 Filtering

Before passing on the output of the XWand to the WorldCursor controller,the yaw and pitch values from the XWand are first filtered to reduce theeffects of noise in the sensors and to ease placing the cursor preciselyon a small target. Two filters are used which average the last n samples(i.e., a box filter). The first is a very slow filter. For example, intested versions of the WorldCursor, this slow filter averagedapproximately the last 2.5 seconds worth of sensor data. This filtertends to dampen most XWand motion and allows the user to move the cursorrelatively slowly for precise positioning. The second filter is muchfaster. For example, in tested versions of the WorldCursor, this fastfilter averaged approximately the last 0.3 seconds worth of sensor data.This fast filter is appropriate for fast XWand movement, when the sensornoise is not as apparent, and responsiveness is desired.

While the user could switch between the filter modes manually via inputto the host computer, the WorldCursor control process running on thatcomputer preferably switches between the slow and fast filtersautomatically. This is accomplished as follows. Referring to FIG. 6, andpresuming that the fast filter is the initial default selection, it isfirst determined if the average speed of the cursor movement has fallenbelow a prescribed slow speed threshold τ_(slow) (process action 600).If not, then the speed of the cursor continues to be monitored byperiodically repeating process action 600. It is noted that the speed ofthe cursor is determined by taking the estimated position returned bythe fast filter, and computing the difference between that estimate andthe same estimate computed in the previous time step, to get speed. Inaddition, the speed can be checked at any appropriate interval thatensures the cursor behaves in the aforementioned controlled manner. Intested embodiments of the control process, the speed was checked everytime step (e,g., about 50 Hz). If it is determined in process action 600that the cursor speed has fallen below the prescribed slow speedthreshold, then in process action 602 the WorldCursor control processswitches to the aforementioned slow filter. The cursor speed is thenmonitored again. More particularly, it is determined if the averagespeed of the cursor movement goes above a prescribed fast speedthreshold τ_(fast) (process action 604). If not, then the speed of thecursor continues to be monitored by periodically repeating processaction 604 in the manner employed previously. In general, the fast speedthreshold τ_(fast) is much higher than the slow speed threshold. Thisprevents frequent shifts and ensures a smooth transition between filtermodes at the appropriate times. While other values can be employed asdesired, in tested embodiments of the control process τ_(slow) was setto 0.05 radians/second and τ_(fast) was set to 0.5 radians/second. If itis determined in process action 604 that the cursor speed has risenabove the prescribed fast speed threshold, then in process action 606the WorldCursor control process switches to the aforementioned fastfilter. At this point, the filter selection procedure continues byrepeating actions 600 through 606, for as long as the WorldCursor systemis activated. In this way, a balance of speed, responsiveness and a finedegree of control are maintained.

3.3.2 Absolute vs. Relative Pointing

In the absolute pointing mode, the WorldCursor's laser ideally points atthe same place as the XWand. The most basic control mapping to computeWorldCursor yaw and pitch (θ_(c), φ_(c)) from the XWand yaw and pitch(θ_(w), φ_(w)) to produce this absolute pointing is,θ_(c)=θ_(c0)+θ_(w)−θ_(w0)   (6)φ_(c)=φ_(c0)+φ_(w)−φ_(w0)   (7)where (φ_(c0), φ_(c0)) and (θ_(w0), φ_(w0)) are offset angles for theWorldCursor and XWand, respectively. These offsets are set to align theorigins of the XWand and WorldCursor in a one time calibrationprocedure.

However, unless the WorldCursor platform and the XWand are very close toone another, it will be impossible to choose offsets (θ_(c0), φ_(c0))and (θ_(w0), φ_(w0)) such that the XWand points directly at theWorldCursor laser spot throughout the range of pointing in the room. Itwill be possible to achieve approximate correspondence for a limitedrange of angles such as one wall of a room, but as soon as theWorldCursor is brought onto the opposite facing wall, for example, thecorrespondence will be far off.

It is not clear if users require even approximate correspondence forsuccessful use of the device. Users' experience with mice suggests thatabsolute correspondence is not necessary. Much in the same way thatusers easily adapt to the relative movement of mouse and cursor, usersof the WorldCursor may be able to adapt to the lack of absolutepointing, subject to the limitation that the laser spot is in theirfield of view. Thus, a relative pointing mode where the WorldCursor'slaser spot moves with the movement of the XWand, but does not point atthe same place in the environment, may be an acceptable operatingcondition. However, a number of procedures can be employed by which theWorldCursor and XWand can be brought into alignment to at leastpartially restore absolute pointing, and obtain other useful informationabout the environment as well. These alignment procedures will now bedescribed in the section below.

3.3.2.1 Clutching

One procedure that the user can employ to re-establish theXWand-WorldCursor correspondence involves a “clutching” technique. Thisis an operation analogous to picking up a computer mouse, moving it inair, and putting the mouse down on the desk again, without the cursormoving. To clutch, the user clicks the XWand button while theWorldCursor control process is in operational mode (as opposed totraining mode). At that point the laser dot stops moving with the XWand.The user may then reorient the XWand, lining it up so that it pointsdirectly at the laser spot. When ready to resume WorldCursor operation,the user clicks the button again and the laser spot resumes moving. Atthis moment new offset values (θ_(c0), φ_(c0)) and (θ_(w0), φ_(w0)) arealso collected.

More particularly, referring to FIGS. 7A and B, the user initiates theoperational mode of the WorldCursor control process, if it is notalready active (process action 700). In the operational mode, thecontrol process normally accepts XWand pitch and yaw inputs, computesthe corresponding pitch and yaw for the WorldCursor laser, and sendscommands to the WorldCursor unit to move the laser to match the computedpitch and yaw values, all in the manner described previously. Thisnormal operation continues until the user presses the XWand button toinitiate the clutching operation (process action 702). Specifically, thecontrol process causes periodic requests to be sent to the XWanddirecting it to provide an orientation message (process action 704), andany incoming orientation message is input (process action 706). Theseorientation messages include the aforementioned XWand pitch and yawvalues that are used to move the laser. They also include the XWandbutton state indicator. Thus, when a message is received it determinedwhether the button indicator indicates that the pointer's button isactivated (process action 708). If not, normal operations are continuedand process actions 706 and 708 are repeated. However, if it isdiscovered that the button state indicator indicates the button has beenactivated, then in process action 710, the control process ceasesproviding movement commands to the WorldCursor device and the laser spotremains stationary in its last position. The user then points the XWandat the location where the laser spot is shining (process action 714) andreleases the button. Any incoming orientation message transmitted by theXWand continues to be input (process action 712) and a determination ismade as to whether the button state indicator included in the messagesfirst indicates that the pointer's button becomes deactivated and thenactivated again, thereby indicating that the user has pushed the XWandbutton again (process action 716). If it has not, process actions 712and 716 are repeated. However, if it has, new offset values for theWorldCursor device and the XWand are collected (process action 718) andnormal operations are resumed (process action 720).

Note that the user may use the foregoing clutching technique not toalign the XWand with the WorldCursor, but to establish a particulardesired range of operation for the XWand. For example, by clutching theuser may set offsets so that the XWand may be held comfortably at theside of the body while the WorldCursor appears on a surface in front ofthe user. The procedure is the same except that the user does not pointthe XWand at the laser spot, but instead orients it as desired. Thisoption put the WorldCursor system into a relative pointing mode asexplained above.

3.3.2.2 Exploiting Room Geometry to Automatically Re-EstablishCorrespondence

Another procedure that the user can employ to establish and maintain areasonable correspondence between the XWand and WorldCursor withoutclutching involves exploiting the geometry of a room in which theWorldCursor system is operating. If the geometry of the room is known interms of a 3D coordinate system, including the position of each wall,the WorldCursor device, and the XWand itself, then the WorldCursor maybe controlled such that the XWand always points directly at the laserspot.

More particularly, if the 3D coordinates of the vertices of the walls inthe room are known, and the 3D position and orientation of the XWand arealso known, it is possible to compute the precise point on the wall thatthe XWand is pointing toward using standard polygonal analysistechniques. Then, using simple trigonometry (i.e., right triangleanalysis in both the yaw and pitch planes), this 3-D ‘wall point’ canthen be related back to the known 3D position of the WorldCursor deviceto compute an updated value of the yaw and pitch (θ_(c), φ_(c)) that isused to point the laser at the aforementioned point on the wall.

By relying on the three dimensional position of the XWand, it may seemlike the very same problem that it is desired to remove in developingthe WorldCursor is being re-introduced—namely the XWand's reliance onexternal 3-D positioning technology. In practice, however, it issufficient to know the room geometry only approximately and stillachieve a useable alignment that requires no clutching. For example, itwill typically suffice to fix the assumed XWand position in the middleof a typical office space, even though this may not coincide with itsactual position. Of course, such an assumption may not provide anaccurate alignment of the XWand and WorldCursor. However, in situationswhere relative pointing is acceptable this assumption can be made.Further other assumed XWand locations could be used instead to make thealignment more accurate. This is particularly useful when it is believedthat the XWand will be at or near a particular position most of the time(e.g., the room occupant's desk). Again, this relies on the fact thatusers tend to be tolerant to constant offsets in alignment similar tothat found in using relative pointing mechanisms such as computer mice.

3.3.2.3 Inferring 3D XWand Position to Re-Establish Correspondence

A more accurate variation of the foregoing room geometry exploitationtechnique that can produce near absolute pointing results involvescombining the geometry exploitation technique with the clutchingprocedure to determine the 3D location of the XWand in the room. If theuser is clutching as described previously so that the XWand points atthe laser spot, each clutching operation provides information that canbe related mathematically to the 3D position of the XWand. After as fewas two clutching operations, it is possible to compute the 3D positionof the XWand.

More particularly, for each clutching operation performed with theWorldCursor laser spot shining on one of the walls whose verticescoordinates are known, there is an associated wall point p_(i), as wellas a vector w_(i) that defines a ray pointing from the XWand to the wallpoint. These values are collected at the end of the clutching operation,after the user has realigned the XWand with the laser spot of theWorldCursor and pressed the XWand button to resume WorldCursor control.Essentially, the wall point p_(i) can be computed via standard polygonalanalysis techniques since the 3D coordinates of the vertices of thewalls in the room are known, as are the 3D position of the WorldCursorbase and orientation of its laser (i.e., the direction the laser ispointing from the base). The vector w_(i) is defined by the orientationof the XWand that is determined as described previously. Assuming theXWand position x is held constant over a number of successive clutchingoperations, thenx+s _(i) w _(i) =p _(i)   (8)where s_(i) is a scalar. It is noted that the “wall points” can actuallybe any point on any surface in the space that is modeled as a polygonvia the procedure described previously.

XWand position x can be found by solving the linear system of equationsgenerated via successive clutching operations using a standard leastsquares approach. A minimum of two clutching operations are required,but for robustness it is be desirable to collect several, particularlyif some of the rays w_(i) are similar, in which case the solution willbe sensitive to small errors in w_(i).

Once the estimate of the XWand position has been updated, the controlprocedure described in Section 3.3.2.2 is used to maintainXWand-WorldCursor correspondence. So long as the actual position of theXWand does not change dramatically, no more clutching operations will benecessary to maintain the correspondence. It is noted that this onlinecalibration requires no more clutching operations than the system whichdoes not exploit the approximate room geometry, and in the long runrequires fewer clutching operations if the user does not move about theroom often. The user would only be aware of the procedure in that aftera while no more clutching operations would be required to keep thecursor and the wand in alignment.

Accordingly, referring to FIG. 8, the XWand-WorldCursor correspondencecan be maintained by first determining if a clutching procedure beenperformed (process action 800). If a clutching operation was notperformed, action 800 is periodically repeated. However, if a clutchingoperation was performed, the 3D location of the point on the wall wherethe WorldCursor laser spot was shining during the clutching operation iscomputed (process action 802). In addition, the vector defining a raypointing from the XWand to the same point on the wall is computed(process action 804). It is next determined if a prescribed number ofclutching procedures have been performed (process action 806). If not,process actions 800 through 806 are repeated. If the prescribed numberis met, then the 3D position of the XWand is computed (process action808). Next, the point on the wall that the XWand is directed toward asit pointed around the space is periodically computed, as is the yaw andpitch values that will direct the WorldCursor laser at that point(process action 810). The WorldCursor laser is directed with these yawand pitch values whenever they are computed (process action 812).Finally, the entire process (i.e., process actions 800 through 812) isrepeated for a long as the WorldCursor is in operation.

It is noted that the number of clutching operations that must beperformed before the XWand-WorldCursor correspondence is updated canalternately be determined based on how different each of theaforementioned rays w_(i) are to each other, rather than using anabsolute number. For example, in one method, if the latest ray to becomputed does not differ from the previous rays computed since the lastcorrespondence update by a prescribed threshold difference, then it isnot counted in terms of how many clutching operations are required. Inthis way, any errors in the computation of the rays will not have asignificant impact on the overall correspondence computations.

It should be noted that some assumptions are made in regard to theforegoing procedures for establishing and maintaining theXWand-WorldCursor correspondence. For example, it is assumed that theuser is clutching to re-establish XWand and WorldCursor correspondence,and not to create a relative pointing relationship between them. Inaddition, it is assumed that the user in not moving about the room, butinstead remains in the same location. It is noted, however, that mostusers are likely to use the XWand from one a small set of staticlocations, much as the TV remote control is likely to be used fromeither the couch or the easy chair. Accordingly, this assumption isquite valid.

3.4 Establishing A 3D Model of the Space Using the WorldCursor

In the foregoing procedures, it was assumed a polygonal model of thespace was available. However, if this is not the case, it is possible toconstruct one given that the positions of the XWand and WorldCursordevices are known. More particularly, given the 3D location of XWand andits orientation (i.e., the direction it is pointing) in combination withthe 3D location of the WorldCursor (which in this coordinate systemcould be designated as the origin if desired) and its orientation (whichis assumed to be directed at the same point as the XWand via theaforementioned clutching operation), then, it is possible to solve forunknown wall points p_(i) using simple trigonometry. Specifically, atriangle is formed by the 3D position of the XWand, the 3D position ofthe Worldcursor device, and the unknown wall point p_(i). Since the 3Dposition of the XWand and the 3D position of the Worldcursor device areknown, the distance between them can be computed. In addition, since theorientation of the XWand and the laser beam are known, it follows thattwo of the angles of the aforementioned triangle are also known. Thus,the wall point pi can be uniquely determined using standardtrigonometric methods. In this way the 3D position of objects anddevices in the room may be learned, much as in the original XWandsystem. For example, the user while holding the XWand in substantiallythe same location within the space would point it at an object (e.g., atits center) or to the vertices of a polygon representing an object ofinterest in the space (including the walls) to establish their 3Dlocation and then enter information about the object, similar to thepreviously described procedure for establishing a model of the space interms of spherical coordinates.

Thus, with a sufficient number of clutching operations to maintain theXWand-WorldCursor correspondence intact, it would be possible to learnthe entire room geometry, including a polygonal model of the walls(which could then be used to thereafter run the foregoing procedures formaintaining the XWand-WorldCursor correspondence more automatically).However, it should be noted that even in this scenario, no moreclutching operations would be necessary than in the case of not using ageometric model.

The foregoing procedure for learning geometry from clutching operationsraises some interesting possibilities for inferring geometry over thenormal use of the device, without relying on a special training mode ora lengthy upfront training session. This has interesting implicationsfor ubiquitous computing applications, many of which rely on geometricmodels to develop location based services, and to reason about userco-presence and proximity to devices. One significant barrier todeploying such systems will be the construction of the necessarygeometric models, especially in the home. The foregoing feature of theWorldCursor system can provide a solution for this problem.

4.0 Applications

4.1 Home Automation

The WorldCursor system is capable of performing all the home 30automation-related tasks of the original XWand system, including turningon and off lights via X10 (i.e., a powerline-based home automationsystem), selecting and manipulating the media player with gestures tocontrol track and volume, and finally selecting and controlling a cursoron a display.

The WorldCursor improves on the XWand system by giving the user muchmore precision in selecting devices. In the original XWand system, forexample, it would be difficult without audio feedback to select one oftwo devices that are located 18 inches apart from 12 feet away. This isdue not only to the limited precision of the XWand system, but also inusers' limited precision in pointing. This uncertainty can be addressedby placing a laser pointer (always on) on the XWand itself, but only upto the precision of the XWand signal processing algorithms.

The WorldCursor precision can be exploited in one application where tocontrol the media player, the user puts the WorldCursor on one ofseveral paper icons (play, pause, next track, previous track) hung onthe wall. The user “presses” the media player button by pushing theXWand button. Menus and other traditional GUI metaphors may also beused.

4.2 Display Surfaces

In the original XWand system, the user may control a cursor on a consoleby first pointing the XWand at the console, and entering a cursorcontrol mode by clicking the XWand button. Exiting the cursor controlmode is accomplished by clicking on a special button presented in theinterface during cursor control mode.

With the precision afforded by the WorldCursor, it is possible toimprove upon this interaction by seamlessly integrating the displaysurface in the world model, without needing to enter a special cursorcontrol mode. For example, once the four corners of the display arespecified with the WorldCursor, the calculations described in Section3.2 may be used to determine if the cursor is on the display. If thecursor is on the display, the laser is turned off, and the prospectiveprojection equations are used to move the cursor to the exact spot onthe display where the laser would be if it had not been turned off. Oncethe user moves the cursor off the display, the cursor is hidden and thelaser is turned back on. Because of the nature of the WorldCursorgeometric model, the registration between the two coordinate systems canbe quite precise.

Further, with networked WorldCursor client processes it will be possibleto extend this functionality across multiple displays simultaneously,moving from display to display seamlessly.

The drawback of the WorldCursor display integration is that the controlof a small display from across the room can be quite difficult, sincethe size of the ‘mousing surface’ is related to the angle subtended bythe display. In the XWand system, this problem is avoided in the specialcursor control mode by using a constant scaled range of angles forcontrol that is independent of where the user is located in the room.One approach to address this limitation is to nonlinearly warp thecoordinate system in the neighborhood of the display.

4.3 Learning Geometric Models

Besides using the WorldCursor to ‘point out’ devices to the system, itmay also be used by the system to ‘point out’ devices to the user. Forexample, if the intelligent environment knew where the user's lost keyswere, the WorldCursor system might direct the user's attention to theirlocation in response to the user's query.

4.4 Non-Veridical Behavior

Thus far WorldCursor control and applications which strive to replicatethe exact motions of the user's manipulations of the XWand have beenexplored. There may also be interesting possibilities in considering howthis one to one mapping may be violated.

For example, in [2] users' ability to finely control a standard laserpointer in an object selection task is studied. In part, performingobject selection and other GUI related tasks with a laser pointer isdifficult because it is surprisingly difficult to hold a laser pointerstill. In Section 3.3.1, a switching filter technique was describedwhich can be used to significantly dampen the motion of the cursor whenthe user is moving the XWand slowly. The WorldCursor can thus do forlaser pointer-based interaction what ‘SteadyCam’ has done for homevideos, by eliminating the jitter and noise associated with standardlaser pointers. This damping filter is especially useful in working withdisplay surfaces, where often GUI elements are small and densely packed.Thus, the WorldCursor system can act as an improved laser pointer.

In an environment where there are many selectable objects, it may beadvantageous to employ a “Snap To” feature for the WorldCursor. If theuser guides the cursor near an active device, a simple spring model maybe used to bring the cursor precisely on the target. For example, asimple distance threshold can be employed such that if the laser beam ismoved to a location near an object that is within the thresholddistance, it is automatically redirected to the shine on that object.Not only may this ease target selection, it also is a convenient way toalert the user to the fact that the object is active and selectable.

In the case when the XWand is used with gestures to control thecurrently selected device, the WorldCursor can be used to ‘play back’the gesture as a way to teach the user the available gestures for thatdevice.

As with the mouse, users of the WorldCursor attend the cursor and almostnever look at the XWand itself. In some cases users have difficultyfinding the laser spot, particularly in the case when the cursor is‘parked’ at a location in the environment out of the field of view, asin the beginning of a clutching operation. One solution to this problemis to bring the cursor to a ‘home’ position after a period ofinactivity, or to animate the laser spot to draw the user's attentionwhen the XWand is first picked up.

While throughout the foregoing description of the WorldCursor system theXWand was employed as a pointing device, this need not be the case.Generally, any conventional pointing device could be used to steer theWorldCursor laser. For example, a computer mouse, track ball, or gamepadcould be adapted to this purpose. Of course, some of the above-describedfeatures of the XWand-WorldCursor combination would not be possible,such as the absolute pointing mode and 3D modeling features, because ofthe lack of pointer orientation data. However, the conventional pointingdevices can support features such as spherical coordinate modeling andobject selection, the above-described display surface feature, andgeneral laser pointer-type functions.

5.0. REFERENCES

-   [1] Haines, E. (1994) In Graphics Gems IV (Ed, Heckbert, P.)    Academic Press, pp. 24-46.-   [2] Myers, B. A., R. Bhatnagar, J. Nichols, C. H. Peck, D. Kong, R.    Miller, and A. C. Long (2002), Interacting At a Distance: Measuring    the Performance of Laser Pointers and Other Devices, in Proceedings    CHI Minneapolis, Minn.-   [3] Zisserman, H. R. a. A. (2000) Multiple View Geometry in Computer    Vision, Cambridge University Press.

1-13. (canceled)
 14. A computer-implemented process for selecting anobject within a space by a user causing a laser beam to be shone on theobject, comprising using a computer to perform the following processactions: directing a laser beam generated by a cursor device comprisinga laser which is capable of projecting a laser beam into the space toshine on said object; determining whether the laser beam is being shoneon the object; and whenever the laser beam is being shone on the object,selecting that object for future control actions. 15-56. (canceled) 57.A computer-readable medium having computer-executable instructions forperforming the process actions recited in claim
 1. 58. A system forselecting an object within a space by a user causing a laser beam to beshone on the object, comprising: a cursor device comprising a laserwhich projects a laser beam into the space; a general purpose computingdevice which is in communication with the cursor device; and a computerprogram comprising program modules executable by the computing device,wherein the computing device is directed by the program modules of thecomputer program to, direct the laser beam to shine on the object;determine whether the laser beam is being shone on the object; andwhenever the laser beam is being shone on the object, selecting thatobject for future control actions.
 59. A computer-implemented processfor selecting an object within a space by a user causing a laser beam tobe shone on the object, comprising: a directing means for directing alaser beam generated by a cursor device comprising a laser which iscapable of projecting a laser beam into the space to shine on saidobject; a determining means for determining whether the laser beam isbeing shone on the object; and whenever the laser beam is being shone onthe object, employing a selecting means for selecting that object forfuture control actions.